PublishersREGULARIZATION TOOLS FOR TRAININGFEED - FORWARD NEURAL NETWORKSPART I : Theory

We present regularization tools for training small, medium and large feed-forward artiicial neural networks. The determination of the weights leads to very ill-conditioned nonlinear least squares problems and regularization is often suggested to get control over the network complexity, small variance error, and to get a nice optimization problem. The algorithms proposed explicitly use a sequence of Tikhonov regularized nonlinear least squares problems. The Gauss-Newton method is applied to the regularized problem that is much less ill-conditioned than the original problem, and exhibits far better convergence properties than a Levenberg-Marquardt method. Numerical results presented connrm that the proposed implementations are more reliable and eecient than the Levenberg-Marquardt method implemented in the Matlab Neural Network toolbox. The proposed algorithms are tested using benchmark problems and guidelines by Lutz Prechelt in the Proben1 package. All software is written in Matlab and gathered in a toolbox.